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Preface 


Today’s  defense  environment  is  placing  growing  pressure  on  defense 
policymakers  to  be  nimble  and  adaptive,  particularly  with  respect  to 
acquisition  systems  and  processes.  This  occasional  paper  is  one  in  a 
series  drawing  upon  the  expertise  of  core  RAND  Corporation  staff 
to  explore  issues  and  offer  suggestions  on  topics  that  are  likely  to  be 
of  critical  importance  to  the  new  leadership:  the  use  of  competition, 
development  of  novel  systems,  prototyping,  risk  management,  organi¬ 
zational  and  management  issues,  and  the  acquisition  workforce.  The 
papers  are  designed  to  inform  new  initiatives  for  markedly  improving 
the  cost,  timeliness,  and  innovativeness  of  weapon  systems  that  the 
Department  of  Defense  (DoD)  intends  to  acquire. 

The  Office  of  the  Secretary  of  Defense  (OSD)  requires  review  of 
Major  Defense  Acquisition  Programs  and  decisions  by  senior  officials 
on  the  basis  of  a  program’s  dollar  value,  irrespective  of  risk.  In  this 
paper,  we  propose  a  new  paradigm  in  which  the  level  of  management 
and  oversight  would  be  based  on  the  level  of  risk  a  program  represents, 
including  technical,  system  integration,  design,  production,  and  busi¬ 
ness  innovation  risk.  We  also  examine  the  extent  to  which  DoD  is 
prepared  to  assess  these  categories  of  risk  and  identify  descriptive  lev¬ 
els  that  could  be  used  to  assess  and  categorize  design  and  business  pro¬ 
cess  risk. 

This  study  was  sponsored  by  the  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics  (OUSD(AT&L)) 
and  conducted  within  the  Acquisition  and  Technology  Policy  Cen¬ 
ter  of  the  RAND  National  Defense  Research  Institute,  a  federally 
funded  research  and  development  center  sponsored  by  the  Office  of 
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the  Secretary  of  Defense,  the  Joint  Staff,  the  Unified  Comhatant  Com¬ 
mands,  the  Navy,  the  Marine  Corps,  the  defense  agencies,  and  the 
defense  Intelligence  Community. 

For  more  information  on  RAND’s  Acquisition  and  Technology 
Policy  Center,  contact  the  Director,  Philip  Anton.  He  can  he  reached 
hy  email  at  atpc-director@rand.org;  hy  phone  at  310-393-0411,  exten¬ 
sion  7798;  or  hy  mail  at  the  RAND  Corporation,  1776  Main  Street, 
Santa  Monica,  California  90407-2138.  More  information  about 
RAND  is  available  at  www.rand.org. 


Dollar  Value  and  Risk  Levels 


Introduction 

Currently,  acquisition  programs  are  grouped  and  then  managed 
at  the  Office  of  the  Secretary  of  Defense  (OSD)  hy  dollar  value — 
depending  on  the  dollar  value,  OSD  provides  different  levels  of 
oversight  and  different  management  processes.  This  approach  has 
been  constantly  refined  over  the  years  without  having  produced  any 
noticeable  improvement  in  terms  of  reducing  the  cost  growth,  sched¬ 
ule  slippage,  and  performance  shortfalls  that  continue  to  plague 
the  acquisition  of  weapon  system  programs.  This  paper  argues  for 
a  different  paradigm:  The  level  of  overall  risk  inherent  in  a  program 
should  be  the  main  basis  for  determining  the  process  and  level  of 
review  a  project  should  receive.' 

Drawing  upon  examples  from  warship  acquisition  programs, 
this  paper  also  argues  that  inadequate  assessment  and  management 
of  various  discrete  program  risks  result  in  adverse  cost,  schedule,  and 
performance  outcomes.  We  examine  existing  scales  for  assessing  some 
of  these  discrete  program  risks  and  make  recommendations  to  better 
assess  and  manage  several  programs  within  the  Defense  Acquisition 
Management  System. 


'  Cost  is  a  factor  that  must  be  considered  when  determining  the  level  of  review.  A  multi¬ 
billion  dollar  program  requires  high-level  review  because  even  a  small  amount  of  cost  growth 
involves  large  dollar  amounts. 
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Managing  by  Risk  Level  Versus  Dollar  Value 


OSD  oversight  is 
based  on  a  program's 
dollar  value, 
irrespective  of  risk. 


Currently,  OSD  requires  review  of  acquisition  programs  and  decisions 
by  senior  officials  on  the  basis  of  a  programs  dollar  value,  irrespective 

_  of  risk,  as  shown  in  Table  1 . 

However,  some  very  costly  projects 
might  have  significantly  less  risk  than  proj¬ 
ects  of  similar  cost,  and  thus  should  require 
less  oversight  as  well  as  the  use  of  different 
criteria  at  milestone  reviews.^  Conversely, 
projects  may  cost  little  but  have  a  lot  of  risk  because  they  tend  to  push 
the  state  of  the  art  in  technology  and  may  also  involve  novel  business  or 
design  processes  that  may  require  more  comprehensive  oversight  than 
just  dollar  value  would  otherwise  indicate.  An  excellent  example  of  this 
type  of  program — the  Advanced  SEAL  Delivery  System  (ASDS) — was 
discussed  in  a  May  2007  report  by  the  U.S.  Government  Accountabil¬ 
ity  Office  (GAO).  The  ASDS  is  a  Special  Operations  Forces’  battery- 
powered  submersible  carried  to  a  deployment  area  by  a  submarine. 
The  operating  parameters  for  the  submersible  required  development 
of  batteries  that  would  push  the  state  of  the  art  in  that  technology. 
The  initial  design,  construct,  and  deliver  contract  was  awarded  for 
$70  million  in  1994  for  delivery  in  1997;  because  of  the  dollar  value. 
Milestone  Decision  Authority  (MDA)  resided  with  the  Navy,  which 
ultimately  accepted  delivery  of  ASDS  in  2003  in  “as  is”  condition  at  a 
cost  in  excess  of  $340  million.  GAO  concluded  that  “Had  the  original 
business  case  for  ASDS  been  properly  assessed  as  an  under-resourced, 
concurrent  technology,  design,  and  construction  effort  led  by  an  inex¬ 
perienced  contractor,  DoD  might  have  adopted  an  alternative  solution 
or  strategy”  (GAO,  May  2007,  p.  13). 


^  For  example,  the  Navy  is  about  to  restart  construction  of  two  DDG  51 -class  destroyers 
at  a  cost  in  excess  of  several  billion  dollars.  Over  60  destroyers  of  this  class  have  already  been 
delivered  or  are  in  the  final  stages  of  construction.  Because  of  this  track  record,  restarting  con¬ 
struction  of  two  new  DDG  51s  will  no  doubt  expose  the  Navy  to  a  far  less  risk  of  adverse  cost, 
schedule,  and  performance  outcomes  than  construction  of  three  multibillion  DDG  1000-class 
ships,  which  are  now  being  designed  and  just  entering  construction. 
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Table  1 

Basis  and  Level  of  Program  Oversight 


Program 

Acquisition 

Category 

(ACAT) 

Basis  for  ACAT  Designation 

Milestone  Decision  Authority  (MDA) 

1 

Estimated  total  expenditure 
for  research,  development, 
test,  and  evaluation  (RDT&E) 
of  more  than  $365  million 
or  for  procurement  of  more 
than  $2,190  billion 

ACAT  ID:  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and 
Logistics 

ACAT  1C:  Head  of  DoD  Component 
(e.g..  Secretary  of  the  Navy)  or,  if  del¬ 
egated,  DoD  component  acquisition 
executive  (e.g..  Assistant  Secretary  of 
the  Navy  for  Research,  Development, 
and  Acquisition) 

II 

Estimated  total  expenditure 
for  RDT&E  of  more  than  $140 
million  or  for  procurement 
of  more  than  $660  million 

DoD  component  acquisition  executive 
or  designate  (e.g.,  program  executive 
officer) 

III 

Does  not  meet  criteria  for 

ACAT  II  or  above;  less  than  a 
MAIS  program 

Designated  by  DoD  component  acqui¬ 
sition  executive  at  the  lowest  level 
appropriate  (e.g.,  program  manager) 

SOURCE:  DoD,  December  8,  2008. 

NOTE:  Estimated  expenditures  are  in  FY  2000  constant  dollars. 


Focusing  on  Causes  Rather  Than  Consequences 

Risk,  or  the  exposure  to  the  chance  of  failure,  is  a  word  heard  fre¬ 
quently  in  the  acquisition  community.  All  acquisition  programs  face 
risk  of  some  form  or  another.  Arguably,  any  new  major  weapon  system 
that  could  he  developed,  produced,  and  fielded  with  no  risk  involved  is 
prohahly  not  worth  acquiring. 

Overtly  or  otherwise,  much  of  a  program  manager’s  time  is  spent 
managing  risk.  After  all,  the  Defense  Acquisition  Management  System, 
shown  in  Figure  1  is,  in  essence,  a  risk-management  process  designed 
to  ensure  success  in  the  timely  delivery  of  weapon  systems  that  meet 
warfighter  requirements  while  staying  within  budget. 
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Figure  1 

The  Defense  Acquisition  Management  System 


User  Needs 


Technology  Opportunities  &  Resources 


•  The  Materiel  Development  Decision  precedes 
entry  into  any  phase  of  the  acquisition 
management  system 

•  Entrance  criteria  met  before  entering  phase 

•  Evolutionary  Acquisition  or  Single  Step  to 
Full  Capability 


A _ 


(Program 

Initiation) 


IOC 


FOC 


Materiel 

Solution 

Analysis 

Technology 
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Engineering  and 
Manufacturing 
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Systems  Acquisition 


Sustainment 


Decision  Point  /\=  Milestone  Review  Decision  Point  if  PDR  is  not  conducted  before  Milestone  B 


SOURCE;  DoD,  December  2,  2008. 


The  risks  most  frequently  mentioned  by  defense  acquisition  offi¬ 
cials  are  cost  growth,  schedule  slippage,  and  performance  shortfalls. 
This  is  not  surprising  as  cost,  schedule,  and  performance  are  the  pri¬ 
mary  attributes  by  which  programs  are  assessed 
for  success  or  failure.  Moreover,  the  Defense 
Acquisition  University  (p.  2)  teaches  that  cost, 
schedule,  and  performance  are  the  risk  factors 
that  program  managers  must  assess  and  man¬ 
age  “to  ensure  that  DoD  is  acquiring  optimum 
systems  that  meet  all  capability  needs.” 

Assessment  of  cost,  schedule,  and  per¬ 
formance  is  clearly  a  management  task,  and 
a  good  program  manager  assesses  these  risks 
using  periodic  data  accumulated  into  management  reports  to  identify 
problems,  regain  lost  ground,  and  then  stay  on  track.  However,  these 
are  broad  measures  of  risk.  A  better  program  manager  proactively  man¬ 
ages  by  using  discrete  program  risks,  submeasures  that  allow  him  or  her 
to  look  ahead  and  act  to  avoid  adverse  cost,  schedule,  and/or  perfor¬ 
mance  trends  and  outcomes.  In  other  words,  managing  by  cost,  sched¬ 
ule,  and  performance  measures  is  akin  to  driving  a  car  while  looking 
solely  in  the  rearview  mirror — it  is  possible,  but  only  if  the  road  stays 


Managing  by 
cost,  schedule 
or  performance 
risks  is  reactive; 
managing 
by  discrete 
programmatic  risks 
is  more  proactive. 
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Straight.  A  better  driver  looks  mostly  out  the  windshield,  with  only  an 
occasional  look  in  the  mirror;  this  driver  anticipates  and  easily  handles 
curves  in  the  road. 

In  this  paper,  we  focus  on  five  discrete  programmatic  risk  cate¬ 
gories: 

•  technical 

•  system  integration 

•  design 

•  production 

•  business. 

Taken  together,  these  risk  categories  portray  overall  acquisition  pro¬ 
gram  risk.^  They  interact  in  numerous  ways  to  affect  a  project’s  cost, 
schedule,  and/or  performance  outcomes:  Obviously,  technologies  that 
do  not  work  affect  performance,  but  so  can  poor  business  decisions 
that  increase  cost  and  lead  to  features  being  deleted  from  the  weapon 
system  to  remain  within  budget. 

The  Defense  Acquisition  Management  System  appears  to  ade¬ 
quately  recognize  that  incorporation  of  new  technologies  into  a 
weapon  system  presents  risk,  providing  metrics  to  systematically  assess 
this  type  of  risk.  Time  is  also  provided  in  the  acquisition  process  for 
system  integration  matters  to  be  identified  and  resolved,  although 
there  is  room  for  improvement.  However,  as  will  be  discussed  in  sub¬ 
sequent  examples,  new  approaches  in  design,  production,  and  busi¬ 
ness  areas  of  acquisition  programs  do  not  appear  to  receive  the  same 
skepticism  and  comprehensive  oversight  that  new  technologies  and 
systems  receive. 

Well-Defined  Process  for  Assessing  Technical  Risk  Is  in  Place 

“Technical  risk”  is  exposure  to  the  chance  that  development  of  critical 
technologies  will  not  meet  program  objectives  within  cost  and  schedule 


^  For  simplicity,  risks  involved  in  fielding,  operating,  and  maintaining  the  ■weapon  system  are 
not  addressed  in  this  paper. 
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The  Defense 
Acquisition  System 
Framework 
incorporates 
assessment  of 
technical  risk  and 
system  integration 
risk  but  puts  less 
emphasis  on  other 
types  of  risk. 


constraints.  In  assessing  technical  risk,  pro¬ 
gram  managers  must  address  the  uncertainty 
in  their  estimates  about  how  much  time  and 
effort  it  will  take  to  make  new  technologies 
work.  The  importance  of  technical  risk  is  well 
understood  in  the  acquisition  community.  For 
example,  DoD  guidance  states  that  “the  man¬ 
agement  and  mitigation  of  technology  risk 
...  is  a  crucial  part  of  overall  program  man¬ 
agement  and  is  especially  relevant  to  meeting 
cost  and  schedule  goals”  (DoD,  2008,  para. 
3.7.2.2). 


Technical  risk  is  also  extensively  addressed  in  the  Defense  Acqui¬ 
sition  Management  System.  The  system  recognizes  evolutionary  acqui¬ 
sition  as  the  preferred  DoD  strategy  for  rapid  acquisition  of  mature 
technology  for  the  user.  One  purpose  of  evolutionary  acquisition  (i.e., 
delivering  capability  in  increments  through  spiral  or  incremental  devel¬ 
opment)  is  to  provide  time  to  better  manage  technology  risk  and  avoid 
adverse  cost  and  schedule  outcomes  that  often  result  from  trying  to 
achieve  difficult  requirements  in  one  step. 

DoD  has  also  established  a  well-dehned  process  based  on  Tech¬ 
nology  Readiness  Levels  (TRLs)  to  categorize  technical  risk  and  help 
ensure  that  key  decisionmakers  understand  the  risk  of  incorporating 
different  technologies  into  weapon  system  acquisition  programs  (the 
TRLs  are  described  in  Table  2).  Using  this  process,  program  offices 
conduct  a  technology  readiness  assessment  under  the  auspices  of  the 
DoD  Component  Science  and  Technology  (S&T)  Executive;  the  Dep¬ 
uty  Under  Secretary  of  Defense  (S&T)  evaluates  the  technology  readi¬ 
ness  assessment  and  forwards  findings  to  the  Overarching  Integrated 
Product  Team  leader  and  Defense  Acquisition  Board. 

The  TRLs  are  a  good  proxy  measurement  for  technical  risk:  The 
lower  the  readiness  level,  the  more  development  needed  to  incorpo¬ 
rate  the  technology  into  a  weapon  system;  and  the  more  development 
needed,  the  greater  the  risk.  Overall,  technology  risk  has  been  handled 
fairly  well  in  warship  acquisition  programs,  which  tend  not  to  push  the 
state  of  the  art  in  technology  as  far  as  do  weapons  and  sensors.  A  recent 
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Table  2 

Technology  Readiness  Levels 

Technology  Readiness  Levels 

1.  Basic  principles  observed  and  reported 

2.  Technology  concept  and/or  application  formulated 

3.  Analytical  and  experimental  critical  function  and/or  characteristic  proof  of  concept 

4.  Component  and/or  breadboard  validation  in  laboratory  environment 

5.  Component  and/or  breadboard  validation  in  relevant  environment 

6.  System/subsystem  model  or  prototype  demonstration  in  a  relevant  environment 

7.  System  prototype  demonstration  in  an  operational  environment 

8.  Actual  system  completed  and  qualified  through  test  and  demonstration 

9.  Actual  system  proven  through  successful  mission  operations 

SOURCE:  DoD,  May  2005. 

NOTE:  See  Mankins,  1995. 


example  is  the  USS  Virginia,  which  incorporates  various  new  technolo¬ 
gies^  and  was  still  delivered  within  four  months  of  the  original  schedule 
established  a  decade  earlier  (Casey,  2007) . 

System  Integration  Risk  Is  Assessed,  but  at  Later  Stages 

The  acquisition  community  also  assesses  system  integration  risk,  hut  it 
lacks  effective  tools  to  measure  and  categorize  this  risk  early  in  a  pro¬ 
gram’s  life  cycle.  “System  integration  risk”  is  exposure  to  the  chance  that 
new  and  existing  technologies  being  employed  in  a  weapon  system  may 
not  work  together  and/or  interact  with  operators  and  maintainers  to 
meet  program  objectives  within  cost  and  schedule  constraints.  System 
integration  can  be  an  issue  within  an  individual  acquisition  program 
(e.g.,  when  subsystems  fail  to  interact).  It  can  also  be  an  issue  between 


For  example,  a  nonpenetrating  photonics  mast  versus  a  periscope,  a  DC  electric  system. 
Lightweight  Wide  Aperture  Sonar  Arrays,  et  cetera. 
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acquisition  programs:  Many  programs  develop  capabilities  that  are  a 
component  of  a  larger  warfighting  capability;  individually,  the  compo¬ 
nent  programs  might  appear  to  be  a  low  or  moderate  risk,  but  in  com¬ 
bination  with  other  programs,  the  overall  risk  might  be  much  higher 
due  to  coordination  and  integration  issues.  A  classic  example  occurred 
during  the  Grenada  invasion  when  Army  and  Navy  communications 
systems  did  not  interact  well  during  the  joint  operation. 

System  integration  risk  is  extensively  treated  after  Milestone 
B,  during  the  engineering  and  manufacturing  development  (EMD) 
phase,  at  which  time  a  program  should  demonstrate  system  integra¬ 
tion,  interoperability,  safety,  and  utility  (DoD,  2008,  para.  3. 7. 1.1). 
While  appropriate  attention  is  given  to  system  integration  risk  dur¬ 
ing  this  phase,  this  assessment  occurs  after  the  second  of  three  mile¬ 
stones  in  the  process,  when  programs  have  typically  built  up  so  much 
momentum  that  they  are  difficult  to  stop,  regardless  of  performance  or 
progress.  Early  consideration  of  system  integration  risk — at  Milestone 
A — by  senior  decisionmakers  could  result  in  developing  and  fund¬ 
ing  integration-risk  mitigation  plans  that  could  considerably  improve 
acquisition  outcomes. 

Combat  systems  in  warships  provide  an  example  of  the  problems 
that  arise  when  decisions  are  made  without  adequate  consideration  of 
system  integration  risk.^  For  example,  early  decisions  on  systems  archi¬ 
tecture  and  processing  approaches  made  without  adequate  consider¬ 
ation  of  risk  led  to  cost,  schedule,  and  performance  problems  with 
submarine  combat  systems  for  the  SSN  6881,  SEAWOEF,  and  Austra¬ 
lian  Collins-c\2ss  submarines.  According  to  a  report  for  the  Parliament 
of  Australia  discussing  the  Collins-class  submarine  (Woolner,  2001), 

Of  the  early  decisions  in  the  Collins  program,  the  one  which  was 
to  have  the  most  public  effect  was  that  concerning  the  nature  of 
the  vessels’  Combat  Data  System  (CDS).  It  has  been  the  subse- 


^  A  combat  system  integrates  information  from  sensors,  synthesizes  this  information  for 
combat  commanders,  and  provides  fire  control  solutions  and  guidance  to  weapons. 
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quent  failure  of  this  system  to  meet  its  design  requirements  that 
has  left  the  submarines  with  a  severely  impaired  combat  capability. 

By  the  end  of  1982,  .  .  .[the  Royal  Australian  Navy  (RAN)] 
had  decided  that  the  electronic  combat  systems  of  the  new  boats 
would  be  fully  integrated.  Instead  of  the  then  standard  central 
computer  performing  all  data  analysis,  the  new  submarine  CDS 
would  use  a  data  bus  to  distribute  information  to  a  number  of 
smaller  computer  work  stations. 

The  report  then  goes  on  to  discuss  the  lack  of  appreciation  for  the  risk 
of  switching  to  the  new  integrated  architecture  for  comhat  systems. 

The  RAN  was  not  alone  in  its  ‘grand  folly.  .  .  .The  Australian 
information  technology  (IT)  industry  assured  the  RAN  of  both 
the  feasibility  and  inherent  advantages  of  a  fully  integrated  com¬ 
bat  system  and  of  its  ability  to  contribute  to  such  a  program. 

Moreover,  the  RAN  was  not  the  only  navy  to  think  that  the  future 
of  combat  data  processing  lay  with  fully  integrated  systems.  The 
USN  [U.S.  Navy]  specified  the  same  concept  for  its  [BSY-2]  Inte¬ 
grated  Combat  System  for  the  U.S.  Navy’s  Seawolf  class  nuclear 
attack  submarines.  This  was  an  even  more  costly  failure  than  the 
Collins  CDS,  absorbing  .  .  .  $1.5  billion  [in  U.S.  dollars]  before 
it  was  cancelled.'" 

Tools  for  assessing  system-integration  maturity  earlier  on  have 
been  proposed.  For  example,  Sauser  et  al.  (2008)  have  proposed  a 
System  Readiness  Level  (SRL)  index  that  would  incorporate  the  cur¬ 
rent  TRL  scale  as  well  as  an  Integration  Readiness  Level  (IRL)  scale. 
The  IRL  scale  they  describe  would  use  nine  levels,  which  appear  com¬ 
patible  with  the  widely  used  TRLs  and  appear  to  be  a  good  proxy 


'■  The  original  citation  mistakenly  attributed  this  to  the  BSY-1  program. 
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measurement  of  system  integration  risk.  The  proposed  IRLs  are  listed 
in  Table  3. 

The  Risks  of  Design  Process  Management  Are  Not  Well  Understood 

“Design  risk”  is  exposure  to  the  chance  that  the  weapon  system’s 
design  will  not  result  in  effective  operation  or  be  easy  to  produce.  It 
is  axiomatic  that  a  good  design  is  essential  to  a  weapon  system’s  per¬ 
formance,  but  the  impact  of  design  on  a  weapon  system’s  production 
cost  and  schedule  outcome  is  not  as  well  appreciated.  However,  deci¬ 
sions  made  early  in  the  design  process  quickly  establish  not  only  the 
performance  but  also  the  ease  of  manufacture  and  resultant  cost  of  the 
weapon  system.  While  the  ability  of  the  design  to  operate  effectively 


Table  3 

Integration  Readiness  Levels 

Integration  Readiness  Levels 


1.  An  interface  between  technologies  is  identified  with  sufficient  detail  to  allow  char¬ 
acterization  of  the  relationship. 

2.  There  is  some  level  of  specificity  to  characterize  the  interaction  (i.e.,  ability  to  influ¬ 
ence)  between  technologies  through  their  interface. 

3.  There  is  compatibility  (i.e.,  a  common  language)  between  technologies  to  orderly 
and  efficiently  integrate  and  interact. 

4.  There  is  sufficient  detail  in  the  quality  and  assurance  of  the  integration  between 
technologies. 

5.  There  is  sufficient  control  between  technologies  necessary  to  establish,  manage, 
and  terminate  the  integration. 

6.  The  integrating  technologies  can  accept,  translate,  and  structure  information  for 
their  intended  application. 

7.  The  integration  of  technologies  is  verified  and  vaiidated  with  sufficient  detail  to  be 
actionable. 

8.  Actual  integration  is  completed  and  mission  quaiified  through  test  and  demonstra¬ 
tion  in  the  system  environment. 

9.  Integration  is  mission  proven  through  successful  mission  operations. 


SOURCE:  Sauser  et  al.,  2008. 


Dollar  Value  and  Risk  Levels  13 


can  be  considered  a  subset  of  technical  risk,  a  more  holistic  approach 
is  for  a  program  manager  to  assess  the  chance  that  the  design  process 
to  be  employed  for  the  weapon  system  will  generate  an  effective,  easy- 
to-produce  weapon. 

The  design  process  necessary  for  an  effective  and  producible 
weapon  system  involves  complex  interactions  between  designers,  sup¬ 
pliers,  production  experts,  planners,  and  estimators.  Design  process 
complexity  has  also  increased  with  the  availability  of  more  sohpisti- 
cated  design  tools  such  as  electronic  product  models  and  computa¬ 
tional  techniques  (e.g.  finite  element  analysis). 

Outcomes  from  two  current  acquisition  programs — the  United 
Kingdom’s  ASTUTE-class  submarine  and  the  U.S.  Navy’s  LPD 
17-class  of  amphibious  transport  dock  ships — demonstrate  why 
senior  decisionmakers  in  the  OSD  acquisi¬ 
tion  process  need  to  better  understand  the 
risks  new  design  processes  and  tools  present. 

The  ASTUTE  was  the  first  UK  submarine 
to  be  designed  through  use  of  an  electronic, 
three-dimensional  computer  product  model. 

The  prime  contractor’s  inability  to  manage 
this  new  process  resulted  in  extensive  delays 
when  design  products  needed  to  build  the 
ship  were  late.  General  Dynamics  ultimately 
had  to  be  hired  to  augment  and  manage  the  final  stages  of  the  sub¬ 
marine’s  detail  design  process.  Because  of  design  and  other  prob¬ 
lems,  the  ASTUTE  program  has  overrun  cost  greatly  and  is  years 
behind  schedule. 

With  EPD  17,  the  U.S.  Navy  competed  the  design  and  pro¬ 
duction  of  the  first  three  ships  of  the  class  using  as  major  evaluation 
and  award  criteria  (1)  the  plans  for  accomplishing  detail  design  and 
other  functions,  (2)  Integrated  Product  Data  Environment  (IPDE) 
tools,  and  (3)  life-cycle  cost  projections;  these  criteria  were  given  more 
weight  than  price  (Comptroller  General  of  the  United  States,  1997). 
The  then-Avondale  Shipyard  in  New  Orleans,  Eouisiana,  partnered 
with  a  firm  that  was  developing  a  new  ship  design  IPDE  tool  and  won 


Two  current 
acquisition 
programs  illustrate 
why  better  insight 
is  needed  into 
the  risks  of  new 
design  processes 
and  tools. 
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the  competition.  Subsequently,  the  LPD  17  experienced  considerable 
cost  growth  (about  70  percent)  and  schedule  delays  (Congressional 
Research  Service,  2008,  p.  12).  GAO  attributed  much  of  this  cost 
growth  to  the  new  design  tool  (GAO,  July  2007): 

In  the  LPD  17  program,  the  Navy’s  reliance  on  an  immature  design 
tool  led  to  problems  that  affected  all  aspects  of  the  lead  ship’s  design. 
Without  a  stable  design,  work  was  often  delayed  from  early  in  the 
building  cycle  to  later,  during  integration  of  the  hull.  Shipbuilders 
stated  that  doing  the  work  at  this  stage  could  cost  up  to  five  times 
the  original  cost.  The  lead  ship  in  the  LPD  class  was  delivered  to 
the  warfighter  incomplete  and  with  numerous  mechanical  failures. 

Senior  decisionmakers  should  require  a  program  manager  pro¬ 
posing  to  use  new  design  processes,  tools,  or  organizations  to  design 
a  weapon  system  to  justify  selection  of  the  new  process,  tool,  or  orga¬ 
nization  and  develop  an  appropriate  risk  mitigation  plan.  An  example 
of  a  design  process  mitigation  plan  comes  from  the  VIRGINIA-class 
submarine  program.  Prior  to  VIRGINIA-class  construction  using  a 
new  Integrated  Product  and  Process  Development  (IPPD)  approach. 
Electric  Boat  started 

a  representative  section  of  the  ship  about  a  year  early  with  a  portion 
of  that  section  started  about  two  years  early.  This  early,  controlled, 
closely  monitored  ship  construction  effort  ensured  thorough 
preparation  for  full-ship  application  and  high  confidence  in  the 
new  process.  (General  Dynamics  Electric  Boat,  2002,  p.  33) 

Evaluation  of  Production  Risks  Lacks  Rigor 

An  earlier  and  more  rigorous  evaluation  of  production  risks  could  save 
DoD  much  difficulty  and  taxpayers  a  lot  of  money.  “Production  risk”  is 
exposure  to  the  chance  that  the  facility,  labor,  manufacturing  processes, 
and  procedures  will  fail  to  produce  the  weapon  system  within  time  and 
cost  constraints.  Producibility — or  “production  capability” — is  a  func¬ 
tion  of  the  design;  production  facilities;  management  skills,  processes. 
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and  experience;  and  workforce  skills  and  experience.  DoD  requires 
assessment  of  contractors’  production  capability  before  production 
contract  award  in  the  production  and  deployment  phase,  but  this  may 
be  too  late  because,  at  this  point,  production  may  be  locked  in  by  the 
organization  that  won  the  design  contract.  Moreover,  in  the  authors’ 
experience  and  as  exemplified  in  the  LPD  17  source-selection  criteria 
discussed  earlier,  the  production  category  of  risk  does  not  receive  the 
same  emphasis  in  selecting  a  shipbuilder  as  other  factors,  such  as  design 
concepts,  past  performance,  and  estimated  cost. 

The  Navy’s  DD  963-class  of  destroyers  and  LHA  1 -class  of 
amphibious  assault  ships  are  classic  examples  of  programs  in  which 
DoD  considered  design  and  production  risk  acceptable  when  award¬ 
ing  contracts,  but  which  went  on  to  experience 
about  the  worst  of  every  production  factor 
possible.  These  ships  presented  little  technical 
and  system  integration  risk,  but  ended  up  far 
behind  schedule  and  over  cost  due  in  part  to 
identifiable  production  risks.  Contracts  were 
awarded  to  the  lowest  bidder,  Litton  Indus¬ 
tries,  which  owned  the  Ingalls  shipyard  in  Pas¬ 
cagoula,  Mississippi.  In  the  late  1960s,  Litton 
Industries  decided  to  invest  in  an  expansion  of 
design  and  production  facilities  for  warships,  building  a  new  shipyard 
on  the  west  bank  of  the  Pascagoula  River,  across  from  its  existing  ship¬ 
yard.  The  new  shipyard  was  designed  to  be  operated  with  a  new  pro¬ 
duction  control  system  using  modular  techniques  for  building  ships 
(Northrup  Grumman,  2008). 

After  the  award  of  the  LHA-  and  DD  963-class  contracts  to  Ingalls 
for  nine  LHAs  and  30  DD  963s  in  the  late  1960s,  Ingalls’  management 
decided  to  shift  construction  of  some  commercial  container  ships  from 
the  old,  conventional  yard  to  the  new  facility  (Northrup  Grumman, 
2008).  The  expectation  was  that  doing  so  would  allow  the  new  facility 
to  start  up  and  have  any  problems  worked  out  while  the  LHA  and  DD 
963  were  being  designed.  However,  production  of  the  container  ships 
using  the  new  control  system  led  to  delays;  consequently,  the  ships 


Acquisition 
programs  can  end 
up  far  behind 
schedule  and 
over  cost  when 
production  risks 
are  not  adequately 
assessed. 
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were  occupying  facilities  and  using  manpower  needed  to  start  produc¬ 
tion  of  the  LHAs  and  DD  963s.  Production  of  the  LHAs  and  DD 
963s  fell  far  behind  and,  in  comhination  with  other  problems  (design- 
related  issues,  inflation,  etc.),  the  costs  were  overrun  substantially  and 
the  ships  were  late  (GlobalSecurity.org,  2008). 

A  greater  emphasis  on  evaluating  production  risks  could  have 
saved  an  enormous  amount  of  time  and  money,  but  the  promised  cost 
savings  resulting  from  construction  in  a  new  state-of-the-art  ship  fabri¬ 
cation  and  assembly  facility  proved  too  good  to  be  true.  The  assessment 
that  the  facility  would  be  derisked  by  building  container  ships  first 
turned  out  to  be  wrong,  and  meanwhile,  two  entire  classes  of  ships  had 
been  priced  and  placed  under  contract. 

A  promising  approach,  initiated  by  the  Missile  Defense  Agency, 
may  provide  program  offices  across  DoD  with  better  insight  about 
production  risk.  The  agency  extended  the  notion  of  TRLs  to  engineer¬ 
ing  and  manufacturing  by  developing  Engineering  and  Manufacturing 
Readiness  Levels  (EMRLs)  to  assess  the  maturity  of  a  program’s  design, 
related  materials,  tooling,  test  equipment,  manufacturing,  quality,  and 
reliability  levels.  There  are  five  EMRLs,  as  shown  in  Table  4. 


Table  4 

Engineering  and  Manufacturing  Readiness  Levels 

Engineering  and  Manufacturing  Readiness  Levels 


1.  System,  component,  or  item  validation  in  laboratory  environment  or  initial  relevant 
engineering  application  or  breadboard,  brass-board  development. 

2.  System  or  components  in  prototype  demonstration  beyond  breadboard,  brass- 
board  development. 

3.  System,  component,  or  item  in  advanced  development.  Ready  for  low-rate  initial 
production. 

4.  Similar  system,  component,  or  item  previously  produced  or  in  production.  System, 
component,  or  item  in  low-rate  initial  production.  Ready  for  full-rate  production. 

5.  Identical  system,  component,  or  item  previously  produced  or  in  production.  System, 
component,  or  item  in  full-rate  production. 


SOURCE:  DoD,  2005. 
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The  Risk  of  Early  Business  Decisions  Is  Not  Fully  Appreciated 

Business  decisions  made  early  in  a  program’s  life  can  significantly  affect 
cost,  schedule,  and  performance  outcomes.  “Business  risk”  is  exposure 
to  the  chance  that  the  overall  acquisition  strategy  for  a  program  will 
not  result  in  the  desired  cost,  schedule,  and/or  performance  outcomes. 
Decisions  about  the  process  to  select  who  will  huild  the  weapon  system, 
the  standards  to  which  it  will  he  huilt,  and  the  schedules  for  designing 
and  building  it  all  entail  risk  that  is  not  always  appreciated  up  front.  To 
evaluate  business  risk,  program  managers  should  assess  the  following: 
(1)  the  extent  to  which  the  acquisition  strategy  can  result  in  selection 
of  the  most  effective,  efficient  design  and  most  effective,  efficient  pro¬ 
duction  entities;  (2)  whether  cost  estimates  and  schedules  are  valid;  (3) 
whether  proper  government  oversight  organizations  are  in  place;  and 
(4)  whether  project  personnel  with  proper  training  and  experience  are 
available. 

A  good  example  of  early  business  decisions  gone  bad  is  the  Navy’s 
Littoral  Combat  Ship  (LCS)  Program.  The  lead  ship,  USS  Freedom 
(LCS  1),  was  recently  delivered  after  experiencing  substantial  cost  over¬ 
runs  and  delivery  delays.  In  congressional  testimony  given  to  explain 
these  outcomes,  the  U.S.  Navy  (2007)  identified  the  following  tenets 
of  the  new  business  model  used  to  acquire  the  LCS: 

•  Construction  of  LCS  seaframes  in  midtier  shipyards  that  “per¬ 
form  predominately  commercial  work,  maintaining  business  pro¬ 
cesses  and  overhead  structures  that  keep  them  competitive  in  the 
world  market”  (i.e.,  little  warship  experience).^ 

•  “A  rapid  24-month  build  cycle  for  each  seaframe,  as  opposed 
to  the  five  or  more  years  that  have  become  the  norm  in  naval 
shipbuilding.” 

•  “The  LM  lead  ship  detail  design  and  construction  effort  was  initi¬ 
ated  simultaneously  and  the  lead  ship  commenced  construction 


^  To  better  understand  the  differences  between  military  and  commercial  shipbuilding,  see 
Birkler  et  ah,  Dijferences  Between  Military  and  Commercial  Shipbuilding:  Implications  for  the 
United  Kingdom’s  Ministry  of  Defence,  Santa  Monica,  Calif:  RAND  Corporation,  MG-236- 
MOD,  2005. 
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only  seven  months  after  the  start  of  final  design  (i.e.,  concurrent 
design/huild) 

•  “In  order  to  address  the  challenges  of  technical  authority  under 
this  environment  (reduction  in  NAVSEA  technical  personnel),  in 
February  2003,  NAVSEA  and  PEO  Ships  made  two  joint  deci¬ 
sions.  The  first  was  to  work  with  the  American  Bureau  of  Ship¬ 
ping  (ABS)  to  develop  a  set  of  standards  (Naval  Vessel  Rules)  that 
could  he  applied  to  non-nuclear  naval  combatant  ships.  The  sec¬ 
ond  was  to  utilize  ABS  to  class®  both  ECS  and  DDG  1 000  using 
the  new  rules.” 

No  doubt  there  were  good  arguments  for  these  individual  pro¬ 
gram  tenets.  However,  the  cumulative  effect  of  the  risks  involved  in 
building  a  new  design  warship  in  small  commercial  shipyards  with 
little  warship  experience  during  a  rapid,  concurrent  design/build  pro¬ 
cess  and  to  a  set  of  technical  standards  themselves  under  development 
appears  to  have  been  greatly  underappreciated.  In  that  same  congres¬ 
sional  testimony,  the  Navy  identified  cost  drivers  for  ECS  1  as  “concur¬ 
rent  design-and-build  while  incorporating  Naval  Vessel  Rules  (NVR), 
reduction  gear  delays  created  by  a  manufacturing  error,  and  insufficient 
program  oversight”  (U.S.  Navy,  2007).  The  risks  inherent  in  utilizing 
an  entirely  new  business  model  to  acquire  warships  were  obviously  nei¬ 
ther  adequately  assessed  nor  managed. 

One  way  to  avoid  such  risk  would  be  to  require  program  manag¬ 
ers  proposing  new  and/or  radical  business  models  to  fully  justify  why 
the  new  model  is  superior  to  past  practice,  recommend  more  frequent 
assessment  points  than  now  required  by  the  Defense  Acquisition  Man¬ 
agement  System,  and  incorporate  exit  strategies  in  contracts  for  the 
government  to  use  if  the  program  fails  to  meet  expectations. 


®  The  American  Bureau  of  Ships  is  known  in  the  commercial  shipping  industry  as  a  “clas¬ 
sification  society,”  which  is  an  organization  that  sets  standards  for  design  and  construction  of 
vessels  and  integral  machinery  within. 
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Conclusions 

The  Defense  Acquisition  System  Framework  has  sufficient  tools  and 
allows  time  for  proper  assessment  and  management  of  technical  risk 
and,  to  some  extent,  of  system  integration  risk.  However,  design,  pro¬ 
duction,  and  business  risks  are  not  always  adequately  assessed  and  man¬ 
aged.  As  shown  in  this  discussion,  scales  exist  that  represent  good  proxy 
measurements  of  technical,  systems  integration,  engineering,  and  pro¬ 
duction  risks;  what  is  missing  are  descriptive  levels  that  could  be  used 
to  assess  and  categorize  design  and  business  process  risk.  We  recom¬ 
mend  that  DoD  explore  establishing  such  levels  and,  in  Tables  5  and 
6,  offer  starting  points  for  doing  so  (based  on  the  authors’  experience), 
which  may  help  program  managers  more  carefully  consider  these  risks. 

In  addition,  we  recommend  the  following  actions  to  better  assess 
and  manage  program  risk  overall: 

•  Assess,  categorize,  and  individually  review  each  technical,  system, 
design,  production,  and  business  risk  of  a  program  at  each  mile¬ 
stone  in  the  Defense  Acquisition  Management  Framework. 

•  Require  program  managers  to  justify  new  or  radical  approaches  to 
design,  production,  or  business  processes  and  develop  and  imple¬ 
ment  risk  mitigation  plans  and/or  contract  off-ramps. 


Table  5 

Proposed  Design  Process  Levels 

Design  Processes 


1.  New,  unproven  processes.  New  design  tools  under  development.  New  design 
organization. 

2.  Large  expansion  of  existing  design  organization.  Many  new  designers  and  supervi¬ 
sors  unfamiliar  with  design  tools  and  processes. 

3.  Existing  design  organization  using  radically  changed  design  tools,  processes,  and/or 
technologies. 

4.  Experienced  design  organization  using  new  design  tools  with  proven  processes. 

5.  Experienced  design  organization  using  existing,  proven  design  tools  and  processes. 
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Table  6 

Proposed  Business  Process  Levels 

Business  Processes 


1.  Using  a  new,  unproven  approach  to  source  selection.  Encouraging  new  sources 
of  supply.  Acquiring  new  technologies  without  well-established  cost-estimating 
relationships.  Requiring  new  government  and/or  contractor  organizations  to  be 
formed. 

2.  Using  new  procurement  process  in  established  industry.  Cost-estimating  rela¬ 
tionships  only  at  high  levels.  Requires  expansion  of  government  and  contractor 
organizations. 

3.  Evolutionary  change  from  prior  acquisition  strategies.  Good  cost-estimating  rela¬ 
tionships.  Existing  government  and  contractor  organizations  can  easily  adapt  to 
changes. 

4.  Using  same  approach  to  buying  similar  products.  Well-established  cost-estimating 
relationships  exist.  Experienced  government  and  contractor  organizations  involved. 

5.  Acquiring  more  of  what  has  been  successfully  bought  before.  Using  the  same  con¬ 
tractor  and  government  organizations. 


Although  such  tools  would  enhance  the  ability  of  program  offices 
to  assess  and  manage  risk,  DoD  should  also  consider  changes  in  over¬ 
sight.  As  stated  at  the  outset  of  this  paper,  the  current  acquisition  system 
requires  review  and  decisions  hy  senior  ofEcials  on  the  basis  of  a  programs 
dollar  value,  irrespective  of  risk.  A  better  use  of  their  limited  time  may  be 
to  focus  on  programs  with  high  risks,  letting  less-senior  officials  deal  with 
lower  risk  programs,  regardless  of  dollar  value.  For  example,  DoD  could 

•  lower  the  MDA  level  for  future  milestones  down 

— two  levels  for  programs  with  low  risk  in  all  risk  categories^ 

— one  level  for  programs  with  moderate  risk  in  all  risk  categories'® 

•  continue  to  follow  the  patterns  for  decision  authority  as  estab¬ 
lished  in  the  Defense  Acquisition  Management  System  for  any 


’  Determination  of  what  constitutes  “low  risk”  is  obviously  subjective.  For  our  purposes, 
“low  risk”  would  be  technology  and  integration  levels  8  and  9  and  EMRL,  design,  and  busi¬ 
ness  levels  4  and  5 . 

For  our  purposes,  “moderate  risk”  would  be  TRL  and  IRL  5  and  6  and  EMRL  design  and 
business  levels  3. 
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program  with  greater  than  moderate  risk  in  any  of  the  hve  catego¬ 
ries  of  program  risk. 

In  this  way,  senior  decisionmakers  might  he  able  to  better  con¬ 
centrate  their  limited  time  on  the  real  potential  problem  areas  in  a 
program  before  problems  occur,  and  direct  actions  to  be  taken  to  avoid 
and/or  mitigate  potential  problems. 
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